Search AI Products and News

In geoscience, geological maps are crucial tools for understanding the Earth's surface and subsurface structures. However, interpreting these complex diagrams requires specialized knowledge and extensive experience. To enhance intelligence in this field, Microsoft Research Asia recently introduced GeoMap-Bench, a new benchmark dataset for evaluating the performance of multimodal large language models (MLLMs) in understanding geological maps. The launch of GeoMap-Bench marks a significant step forward in AI applications for geological map interpretation. Microsoft researchers, in collaboration with...

Driven by multimodal large language models (MLLMs), significant advancements have been made in tasks related to images and videos, including visual question answering, narrative generation, and interactive editing. However, achieving fine-grained understanding of video content still poses major challenges. These challenges involve tasks such as pixel-level segmentation, tracking with language descriptions, and visual question answering based on specific video prompts. Although current state-of-the-art video perception models excel in segmentation and tracking tasks, they still fall short in open language understanding and conversational capabilities.

Recently, the Meta AI team introduced LongVU, a novel spatio-temporal adaptive compression mechanism aimed at enhancing the language understanding capabilities of long videos. Traditional multimodal large language models (MLLMs) face limitations in context length when processing long videos, and LongVU was created to address this challenge. LongVU operates primarily by filtering duplicate frames and employing inter-frame token compression techniques to efficiently utilize context length, allowing it to reduce video data while preserving visual details.

Recently, Apple's AI research team launched their next-generation family of Multimodal Large Language Models (MLLMs) - MM1.5. This series of models can integrate various data types such as text and images, showcasing new capabilities of AI in understanding complex tasks. Tasks like visual question answering, image generation, and multimodal data interpretation can be better addressed with the help of these models. A major challenge for multimodal models is how to achieve effective interaction between different data types. Previous models often struggled in this aspect.

As AI technology continues to advance, understanding user interfaces (UI) has become a key challenge in creating intuitive and useful AI applications. Recently, researchers at Apple introduced UI-JEPA in a new paper, an architecture designed for lightweight, device-side UI understanding that maintains high performance while significantly reducing the computational requirements for UI understanding. The challenge of UI understanding lies in processing cross-modal features, including images and natural language, to capture temporal relationships within UI sequences. Despite the complexity, multimodal large language models...

Recently, an open-source project called gptpdf has gained 1.1k stars on GitHub. It uses a VLLM model similar to GPT-4o to parse PDF files and convert them into Markdown format.gptpdf Product Entry: https://top.aibase.com/tool/gptpdf It is learned that the code for this project consists of only 293 lines, yet it can nearly perfectly parse and include content such as formatting, mathematical formulas, tables, images, charts, and more. The implementation steps of gptpdf are: 1) Use the PyMuPDF libr